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This article presents a mathematical function for measuring the congruence among the strand representation 
in the enacted curriculum, state assessments, and district pacing guides. Using the Manhattan distance as its 
basis, this method allows researchers to incorporate the strand coverage of the enacted curriculum (i.e., the 
curriculum actually experienced by students) into alignment studies even when the analysis of student work 
is limited to short-term collections. The result is a mathematical instrument involving just basic arithmetic that 
even those without specialized knowledge of mathematics can imderstand and apply. Using this instrument, 
educators should monitor the adherence of the enacted curriculum and district pacing guides to the topics 
encompassed on state and district assessments. (Contains 4 figures, 2 tables, and 13 references.) 
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1. INTRODUCTION 

The content taught to students on a daily basis (the enacted curriculum) significantly influences 
how much they learn [Gamoran et al. 1997; Wiley et al. 1995]. Even in cases where a district 
adopts a fully standards-based curriculum, students can struggle on state assessments if the 
enacted curriculum diverges significantly from the adopted curriculum [Firestone et al. 2000; 
Bemauer 1999; Desmond et al. 1998]. 

Using a combination of surveys and collected student assignments. Firestone and colleagues 
[2000] appear to have performed the first large-scale effort to analyze the enacted curriculum. 
Other efforts relied on teacher logs, a form of self-reporting survey [Rowan et al. 2002; Porter 
1997; Porter 1989]. Although more logistically taxing than surveys, private companies have 
collected and directly examined completed student assignments on a large-scale basis 
[The Standards Company LLC 2008] . 
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2. NOMENCLATURE 

State content standards typically categorize among three or four levels. A typical hierarchy 
comprises (from the most general to the most specific): 

1. Subject (e.g., "English language arts") 

2. Strand (e.g., "grammar and punctuation") 

3. Standard (e.g., "Students will use commas properly.") 

Unfortunately, this nomenclature varies from state to state, even to the point that a "strand" in 
one state corresponds to a "standard" in another state. This article adheres to the categorization 
scheme listed above, focusing mostly on level 2 (the strand). 

3. EXTENT OE COVERAGE 

Even when the enacted curriculum perfectly aligns to standards, students can still perform poorly on 
assessments if the curricular content they receive in class fails to adequately cover the range of strands 
they encounter on the state assessment. Eor example, students taught content predominantly from 
grammar/punctuation strands will likely struggle on sections of the state assessment that address 
reading comprehension of informational text. Similarly, students who overly concentrate on basic 
number facts at the expense of algebra will likely struggle on the state mathematics assessment. 
Therefore, educational agencies seeking to raise state test scores must ensure that the enacted 
curriculum adequately samples every tested strand before state testing begins. 

To strengthen adherence between the content taught by teachers on a daily basis and state 
and district guidelines, district staff often create pacing calendars (sometimes called pacing 
guides) that provide a timeline for teaching particular objectives. However, district pacing 
calendars and state assessments do not always match tightly, nor do teachers always follow the 
pacing calendars. 

4. APPLICATION 

As a worthwhile study, researchers and school administrators can compare the strand sampling 
in the enacted curriculum with that found in state assessments to rmcover underemphasis and 
overemphasis of particular strands of the standards. Eortrmately, many states release blueprints 
describing the percentage they weight each strand on their state assessments. In most cases the 
state releases the blueprints in advance of the assessments to help guide lesson plarming. 
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Accurate analysis of the enacted curriculum, however, requires the collection of completed 
student assignments. Typically, such short-term studies span a few days or weeks. Unlike 
alignment and rigor studies, however, researchers would waste time and effort comparing the 
extent-of-coverage of a short-term student assignment collection to that found on the state 
assessment— teachers often, and some may argue should, focus on particular strands for any 
given short period of time. 

Lengthening the collection period would improve this situation, but long-term collections 
can impose tremendous logistical burdens on district staff. The approach used in this article 
offers a remedy— a means of gauging the congruence between the state assessments and 
enacted curriculum with a short-term assignment collection. 

The method presented in this article involves two distinct phases: 

1. A broad-scope comparison between the extent of coverage of the state assessments and the 
district pacing calendar. 

2. A narrow-scope comparison between the enacted curriculum and the portion of the pacing 
calendar that spans the collection period. 

The next section discusses a mathematical means of gauging the strand congruence for both 
steps and a way to summarize the results mathematically into an overall congruence function. We 
must first, however, discuss the mathematics that underlie this approach to measuring congruence. 

5. MANHATTAN DISTANCE 

The Manhattan distance (sometimes called the taxicab distance) obtains its name from the 
shortest distance a vehicle can travel between two points in a city with streets arranged in a 
rectangular grid [Black 2006] . Some readers with military experience will remember the act of 
"squaring corners," which is also directly analogous to traveling along Manhattan paths. Others 
will liken the paths to the possible moves of a rook on a chessboard. 
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Fig. 1. All three paths from Point A to Point B in this figure feafure the same travel distance, termed the 
Manhattan distance. The Euclidean distance ("as the crow flies") corresponds to the dashed line. 



Multiple paths will feature the same Manhattan distance S between two points. Figure 1 
identifies three paths between two points (more exist) that feature the same Manhattan distance. 
The mathematical expression for S for each path all simplify to the same expression 

S = dy + d2 — 5j| + 1^2 ~ H 1- — 5^|), (1) 

where N represents the dimensionality of the system (N = 2 in Figure 1). This expression 
translates to "S is the sum of individual perpendicular-path lengths needed to move from Point 
A to Point B in N dimensions." 

From here on we will scale the lengths of each side of the grid to unity, producing a 
maximum Manhattan distance of 2. (This holds even in higher dimensions, since the distance 
needed to travel to the origin is never larger than 1 and the distance needed to travel from there 
to any other point is also never larger than 1.) As a result, our approach relies on the scaled 
Manhattan distance s = (1/2)S to measure congruence. 

6. BROAD-SCOPE INDEX 

This article defines the broad-scope index, Cpt, as the congruence between the state assessment 
and the district pacing calendar. First, let T = (Ti, Ti, Ta,... Tn) denote the percentage of questions 
on the state assessment that correspond to each of the N strands of the state content standards. 
For example, T = (0.45, 0.50, 0.05) indicates that 45% of the state assessment corresponded to the 
first strand, 50% to the second strand, and 5% to the third strand. Likewise, P = (Pi, Pi, Ps,... Pn) 
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represents the percentage of questions found on that portion of the pacing calendar that extends 
from the start of the school year to the state testing date. Using the scaled Manhattan distance 
SPT between Sets T and P, the congruence index 

CpT =l--(\P^-T^\ + \P^-T^\ + \P^ -rjl + ’-’ + lP^ ^^iv|) (2) 

measures the congruence between this "tested-year" pacing calendar and the state assessment. Perfect 
congruence correlates to Q^T = 1. At the other extreme, Q^T = 0 represents complete misalignment. 

The broad-scope index performs a meaningful function on its own— low values of the broad- 
scope index indicate that the pacing calendar may need restructuring. 



7. NARROW-SCOPE INDEX 



Consider a typical week-long collection of student assignments. In a vein similar to the previous 
section, p = (pi, pi, pi,... pv) represents the percentage of questions that align to N strands of the 
state content standards found in the pacing calendar, but only for that portion of the pacing 
calendar sparming the collection period. Analogously, e = {ei, ei, ei,... cn) tabulates the 
percentage of questions in the enacted curriculum (that is, the assignments collected for 
analysis) that align to the same N strands. Therefore, 



C =l-5„ 




Pi + F2 



P2 + F3 



P3 +---F1V 




(3) 



This narrow-scope index measures the extent to which the teaching staff adhered faithfully to 
the pacing calendar during the collection week. 

Like the broad-scope index, the narrow-scope index serves a useful purpose in its own 
right— low values indicate that teachers may need to reassess the content they teach in light of 
the content suggested by the pacing calendar. 



8. CONGRUENCE EUNCTION 

We define the overall congruence function that measures the match between the enacted curriculum 
and the state content standards as the product of the broad-scope and narrow-scope indices, 

c = fic/,c;, (4) 

a variation of the familiar Walberg function used to model educational productivity [Walberg 
1980]. The coefficient B normalizes the index, whereas the exponents/' and k weight each 
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contribution to the overall index. Although researchers could establish values for j and k using 
multiple regression, there is little reason to think that one index should dominate over the other, 
so this article assumes j = k = l for the sake of simplicity. Since both indices range from 0 to 1, 
then B = 1 as well. Therefore, 



As a product of two terms, schools can perform well with respect to one index but still 
register a low congruence for the following reasons: 

1. The pacing calendar could align closely to the state assessment, but the enacted curriculum 
does not align to the pacing calendar. In this case, teachers delivered the "wrong" content. 

2. The enacted curriculum could align closely to the adopted pacing calendar, but the pacing 
calendar does not align to the state assessment. In this case, the pacing calendar compelled 
teachers to deliver the "wrong" content. 

Both results would expose a severe misalignment between the content that teachers delivered to 
their students and the questions asked on state assessments. If so, low values of the congruence 
index make sense. One can strengthen this justification for using the product term using an 
analogy to probability theory: If students have (say) learned only half of what they need in 
preparation for a district test because of a misalignment of the enacted curriculum to the district 
pacing calendar, but the pacing calendar itself reflects only half of the strands coverage on the 
state assessment, then the product (50%)(50%) = 25% reasonably describes their chances of 
adequately responding to a problem on the state assessment. 

The congruence function cannot model the situation in which the enacted curriculum aligns 
closely to state assessments, but the pacing calendar matches poorly to both the state assessments 
and the enacted curriculum. In such a situation, teachers apparently used the state content 
standards as their framework for teaching rather than the pacing calendar. (If so, the teaching staff 
could judge the pacing calendar worthless with much justification.) However, if one could 
accurately measure the congruence between the enacted curriculum and the state assessment 
(which again would require a long-term collection of student work), then the direct calculation 



c = c c 



( 5 ) 




K “^1 +F2 “^2 +'" + k 




( 6 ) 



would replace the expressions in Equations 1 and 2. 
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Note that forcing the exponents in Equation (4) to sum to 1 would allow the congruence 
function to follow the Law of Diminishing Returns [Cobb et al. 1928] —for a fixed value of one of 
the indices, the gains of C would diminish as the other index increases. Although this 
functionality seems reasonable from a dimensional analysis standpoint, the authors of this 
article have already argued that the probability model introduced earlier describes the effect of 
subsequent misalignment realistically, so the approach advocated in this article declines to force 
the sum of; and k to equal 1. 



9. ALTERNATIVE FORMULATIONS 



One might consider using the average discrepancy 



S 



ave 




A +F2-;^2 +--- + FV 




( 7 ) 



rather than the Manhattan distance. However, the average discrepancy scales incorrectly and so 
produces considerable error, especially in the limit of complete misalignment. Consider a state 
that tests four strands of the standards: Number sense, algebra, geometry, and measurement. 
Suppose the pacing calendar at (fictitious) Gingko Middle School specifies that teachers should 
address only algebra during a certain week; therefore, p = (0, 1, 0, 0). However, work collected 
from the school during this time features only geometry; so e = (0, 0, 1, 0). In principle, this 
situation constitutes a complete misalignment, but the average discrepancy Cep = 0.50, not 0. 

The Euclidean "as the crow flies" distance (dashed line in Eigure 1) 




A +-" + K-Pv 



( 8 ) 



functions considerably better than the average discrepancy. Eor Gingko Middle School, Cep = 0, 
a reassuring result. However, if some of the work collected from the school aligned to 
measurement instead of geometry. Cep no longer equals 0 even though the extent of coverage 
remains completely misaligned. This inconsistency arises because the shortest distance between 
two points does not truly correlate to congruence. Consider a target point located in three 
dimensions at (1, 0, 0). Physically, Point A = (0, 0.5, 0.5) lies closer to the target than 
Point B = (0, 1, 0) and so would generate a larger Cep index. In terms of congruence, however, 
both represent complete misalignment. 
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10. VISUAL INTERPRETATION 

The Manhattan distance S used to calculate the broad-scope and narrow-scope indices is 
directly analogous to the distance needed to travel along orthogonal (perpendicular) axes 
between two points in N-dimensional space, where N depicts the total number of strands 
appearing in the calculations. 

Consider the two-dimensional case, tabulated in Table 1 and illustrated in Figure 2, where 
the state tests knowledge in just two strands of the state content standards. Here, the enacted 
curriculum reflected only Strand A (Point e), as opposed to the requirements of the pacing 
calendar for the assignment collection period (Point p), which specified that only 75% of the 
content should have sampled Strand A. This represents a total Manhattan distance of 
0.25 + 0.25 = 0.5, so that Cep = 1 - (0.5/2) = 0.75. On the other hand, the overall pacing calendar 
(Point P) specifies that only 25% should sample Strand A; however, 50% of the state test 
(Point T) is composed of Strand A. This also represents the same value for Cpt, that is, Cpt = 0.75. 
Therefore, the congruence C = (0.75)(0.75) = 0.56. 




Fig. 2. Demonstration of the Manhattan distances and their relation to the broad-range and narrow-range 
indices for the simplistic case of only two strands. All points must lie on the x + y =1 line (depicted with 
thin dashes) since all strand percentages must sum to 100%. Ideally, a long-term sampling of student 
work could directly establish Cei (thick dashed line), eliminating the need for Cpt and Cep. 
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Table 1. Sample calculation of Cpt and Cep for two strands. 





Cpt 




Cep 


Strand 


T 


P 


Idl 




E 


p 


Idl 


A 


Ti 


0.50 


Pi 


0.25 


0.25 




ei 


1.00 


Pi 


0.75 


0.25 


B 


Ti 


0.50 


Pi 


0.75 


0.25 




62 


0.00 


Pi 


0.25 


0.25 










Spt 


0.50 










Sep 


0.50 










Spt 


0.25 










Sep 


0.25 










Cpt 


0.75 










Cep 

1 


0.75 



Strand 3 




Strand 2 



Fig. 3. A visual analogy between the Manhattan distance used to measure the congruence between 
curricular materials encompassing three strands of fhe sfafe confenf standards and a metropolitan grid of 
skyscrapers. (For clariy, we only show fhe narrow-scope skyscrapers.) 



Physically, the three-dimensional case (that is, three strands) mirrors a metropolitan grid of 
skyscrapers, with one strand aligned north, another strand aligned west, and the number of 
floors in each skyscraper corresponding to the third strand. The Manhattan distance between 
(say) the enacted curriculum and the weekly pacing calendar would then describe the distance 
needed to descend the elevator in one skyscraper, travel along perpendicular city streets to 
another skyscraper, and ascend the elevator to the target value. (See Figure 3.) 

The 2007 Nevada English language arts standards serve as a good example of the three- 
dimensional model. Table 2 lists the Nevada state assessment blueprint coefficients for eighth- 
grade English language arts [Nevada Department of Education 2008], along with values 
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obtained from a hypothetical school pacing calendar and data obtained from an analysis of 
student assignments over a one-week collection. Note that the broad-scope pacing calendar 
data, which reflect the (roughly) seven months preceding the state test date, do not coincide 
with the narrow-scope pacing calendar data, which reflect a collection period on the order of a 
few days or weeks. Again, the Manhattan distance Spt equates to the perpendicular-path 
distance between Points P and T, illustrated with the thick line in the figure. Since 
Cpt = 1 - (1/2)Spt, as the distance between the two points shrinks, Cpt increases. The same 
reasoning applies to the narrow-scope index. 



Table 2. Sample calculation of Cpt and Cep for a hypothetical Nevada school. 





Cpt 




Cep 


Strand 


T 


P 


Id 1 




e 


P 


Id 1 


Reading 1.0 


Ti 


0.23 


Pi 


0.21 


0.15 




ei 


0.72 


pi 


0.50 


0.22 


Reading 3.0 


Ti 


0.31 


Pi 


0.74 


0.43 




62 


0.28 


pi 


0.50 


0.22 


Reading 4.0 


T3 


0.46 


Ps 


0.18 


0.28 




63 


0.00 


ps 


0.00 


0.00 










Spt 


0.86 










Sep 


0.44 










Spt 


0.43 










Sep 


0.22 










Cpt 


0.57 










Cep 


0.78 



From the data and calculations shown in Table 2, even though the extent of coverage of the 
student assignments looks nothing like that on the state assessment, the school achieves a 
reasonable value for the narrow-scope index because the enacted curriculum mimics to a large 
extent the adopted pacing calendar. However, the pacing calendar does not adequately reflect 
the state assessments, producing a small broad-scope index and, as a result, a small congruence 
(C = 0.47). In terms of curriculum coverage, the low extent of coverage of the enacted curriculum 
at this school did not occur during the collection period, but rather before the year started. 
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Reading 4.0 




Fig. 4. A visual representation of the Manhattan distances Sep and Spt in three dimensions for fhe dafa in 
Table II, wifh each axis corresponding fo a parficular strand of fhe Nevada sfate contenf sfandards. (Some 
of fhe axis labels are omitted for clarity.) All points must lie on the x + y + z = 1 plane (depicted with 
dashes) since all strand percentages must sum to 100%. Note that points P and p do not necessarily 
coincide or even lie near each other. A long-term collection of studenf assignments could establish Sei (the 
length of fhe heavy doffed line); however, shorf collecfion periods require measuring Sep and Spt insfead. 



11. DISCUSSION 

The Manhattan distance provides a strong visual interpretation of congruence. Mathematically, 
the Manhattan distance involves nothing but basic arithmetic, making the approach accessible 
to those without special training in mathematics. As a result, the congruence formulation 
presented in this article offers a practical means for researchers to present research results to 
district coaches and, in turn, for district coaches to inform teachers. 

In some situations, the state only assesses a limited portion of the state content standards. An 
example occurs in California, which does not explicitly test the mathematical-reasoning strand. 
However, a district would probably consider such a standard important to its academic mission 
and include it in its pacing calendar. In this case, the appearance of curriculum related to these 
strands, although within the goals of the district, will artificially lower the congruence. 
Researchers can remedy this situation by simply filtering out any appearance of such non-tested 
standards from the pacing calendar and enacted curriculum before calculating the indices. 

Some districts establish benchmark assessments to identify deficiencies in learning before the 
state assesses their students. Researchers can incorporate the extent of coverage of district 
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benchmarks by defining (1) a broad-scope index between the district pacing calendar and the 
district benchmark and (2) another broad-scope index between the district benchmark and the 
state assessment. Equation 5 then incorporates three indices instead of two. 

Naturally, not all schools or districts provide pacing calendars. However, researchers could 
substitute the adopted textbook for the pacing calendar once they have established the broad- 
scope index describing the congruence between the textbook and the state assessment. (Final 
exams could potentially fill this role as well.) Since the enacted curriculum incorporates all 
curricular materials given to students during a specific collection period, researchers can 
measure the narrow-scope index by comparing the enacted curriculum to that portion of the 
textbook that encompassed the collection period. 
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